Recently several companies have started to research and publish (WO2007082562, a two-image method with a residual layer, WO2005104035, a somewhat similar method in which one can form a ratio image for boosting a low dynamic range (LDR) re-grading of an HDR scene) on how they can encode at least one still picture, or a video of several, so-called high dynamic range images, which HDR images are characterized by that they typically encode or are able to encode at least some object luminances of at least 1000 nit, but also dark luminances of e.g. below 0.1 nit, and are of sufficient quality to be rendered on so-called HDR displays, which have peak brightnesses (being the luminance of the display white, i.e. the brightest renderable color) typically above 800 nit, or even 1000 nit, and potentially e.g. 2000 or 5000 or even 10,000 nit. Of course these images of say a movie may and must also be showable on an LDR display with a peak brightness typically around 100 nit, e.g. when the viewer wants to continue watching the movie on his portable display, and typically some different object luminances and/or colors are needed in the LDR vs. the HDR image encoding (e.g. the relative luminance on a range [0,1] of an object in a HDR grading may need to be much lower than in the LDR graded image, because it will be displayed with a much brighter backlight). Needless to say also that video encoding may have additional requirements compared to still image encoding, e.g. to allow cheap real-time processing, etc.
So typically the content creator makes a HDR image version or look, which is typically the master grading (the starting point from which further gradings can be created, which are looks on the same scene when one needs to render on displays with different peak brightness capabilities, and which is typically done by giving the various objects in an image straight from camera nice artistic colors, to convey e.g. some mood). I.e. with “a grading” we indicate that an image has been so tailored by a human color grader that the colors of objects look artistically correct to the grader (e.g. he may make a dark basement in which the objects in the shadows are hardly visible, yet there is also a single lamp on the ceiling which brightly shines, and those various rendered luminances may need to be smartly coordinated to give the viewer the optimal experience) for a given intended rendering scenario, and below we teach the technical components for enabling such a grading process yielding a graded image also called a grading, given our limitations of HDR encoding. And then the grader typically also makes a legacy LDR image (also called standard SDR image), which can be used to serve the legacy LDR displays, which may still be in the field for a long time to come. These can be transmitted alternatively as separate image communications on e.g. a video communication network like the internet, or a DVB-T channel. Or WO2007082562 and WO2005104035 teach scalable coding methods, in which the HDR image is reconstructable at a receiving side from the LDR image, some tone mapping on it, and a residual HDR image to come sufficiently close to the HDR original. Such a scalable encoding could then be co-stored on a memory product like e.g. a solid state memory stick, and the receiving side apparatus, e.g. a television, or a settopbox (STB), can then determine which would be the most appropriate version for its connected television.
I.e. one stores in one sector of the memory the basic LDR images, and in another sector the HDR images, or the correction images such as luminance boost image from which one can starting from the corresponding LDR images for the same time moments calculate the HDR images. E.g. for televisions up to 700 nit whichever unit does the calculation of the ultimate image to be rendered on the television may use the LDR graded image, and above 700 nit it may use the HDR image (e.g. by polling which display of which PB is connected, or knowing that if the display does the best image selection itself).
Whilst this allows to make two artistically perfect reference gradings of a HDR scene for two specific rendering scenarios, e.g. a 5000 nit television, and an LDR one (which standard has 100 nit PB), little research has been done and published on how one can handle the in-between televisions of peak brightness intermediate the peak brightnesses corresponding to the two artistic grading images which can be retrieved or determined at an image data receiving side (the corresponding peak brightness of a coding being defined as the to be rendered luminance on a reference display when the maximum code, e.g. 1023 for 10 bit, is inputted, e.g. 5000 nit for a 5000 nit graded look) which will no doubt soon also be deployed in the market, e.g. an 1800 nit television. Applicant has done experiments which show that to have a really good, convincing artistic look for any intermediate or out of range display (e.g. obtaining a 50 nit look which is below the lowest grading which may typically be 100 nit), neither the HDR nor the LDR image is really good for that intermediate peak brightness display (which we will herebelow call a medium dynamic range (MDR) display). Also, it could be that the consumer has an actual television or other display present in his living room which is brighter than the peak brightness of the reference display as an optimal intended display for the received HDR image grading, i.e. e.g. 10000 nit vs. 5000 nit, and then it may be desirable to have an improved grading for these higher brightness displays too, despite the fact that the content creator thought it was only necessary to specify his look on the HDR scene for displays up to 5000 nit PB. E.g. applicant found that in critical scenes, e.g. a face of a person in the dark may become too dark when using the HDR image due to the inappropriately high contrast of that HDR image for a lower peak brightness display rendering, yet the LDR image is too bright in many places, drastically changing the mood of e.g. a night scene. FIG. 14 shows an example of a typical HDR scene image handling we want to be able to achieve. 1401 shows the original scene, or at least, how it has been approximated in a master HDR grading (because one will typically not encode the sun as to be rendered on a display on its original 1 billion nit brightness, but rather as e.g. 5000 nit pixels). We see a scene which has some indoors objects, which will be relatively darker, but typically not really dark, e.g. between 1 and 200 nit, and some sunny outdoors objects seen through the window, like the house, which may in real life have luminances of several 1000s of nits, but which for night time indoors television viewing are better rendered around e.g. 1000 nit. In a first grading, which we will call an HDR grading with a peak brightness of the first image in this mere example PB_IM1 corresponding to a HDR peak brightness PB_H of e.g. 5000 nit, we will find it useful to position the indoor objects relatively low on a relative luminance axis (so that on an absolute luminance axis they would be rendered at luminances around 30 nit), and the outside objects would be somewhere around or above the middle of the luminance range, depending on the grader's preferences for this shot in e.g. the movie, or broadcast, etc. (in case of life broadcast the grading may be as simple as tuning only very few parameters prior to going on air, e.g. using a largely fixed mapping between the HDR and LDR look, but adding e.g. a single parameter gpm for enabling the display tuning) Which actual codes correspond to the desired luminances not only depends on the PB of the coding, but also on the shape of the used code allocation function, which is also sometimes called an Opto-electronic conversion or transfer function (OECF; OETF), and which for HDR coding typically has a steep shape, steeper than the gamma 1/2.2 functions of LDR (the skilled person will understand that one can formulate the technologies in either representation, so where for simplicity we herebelow elucidate our concepts in luminance representations, at least some of the steps may be mutatis mutandis be applied in lumas, i.e. the e.g. 10 bit codings of the luminances).
One needs a corresponding LDR look (in this mere example called IM_GRAD_LXDR), for which of course all the various objects of the larger luminance dynamic range have to be squeezed in a smaller dynamic range, corresponding to a 100 nit PB. The grader will define color transformation strategies, typically simple functions to keep the video communication integrated circuits simple at least for the coming years, which define how to reposition the luminances of all objects (e.g. as can be seen one would need to position the house close to the maximum of the luminance range and corresponding code range to keep it looking sufficiently bright compared to the indoors, which may for certain embodiments e.g. be done with a soft-clipping tone mapping). This is what is specified on the content creation side, on a grading apparatus 1402, and a content using apparatus may need to determine, based on the information of the graded looks (S_im) which it receives over some image communication medium (1403), which optimal luminances the various objects should have for an actual display having a peak brightness which is unequal to the peak brightness corresponding to any of the typically two received artistical gradings (or at least data of those images). In this example that may involve various strategies. E.g., the dark indoors objects are well-renderable on displays of any PB, even 100 nit, hence the color optimization may keep them at or near 30 nit for whatever intended PB. The house may need to get some to be rendered luminance in between that of the LDR and HDR gradings, and the sun may be given the brightest possible color (i.e. PB) on any connected or to be connected display.
Now we want to emphasize already, as will become clear later, that we have developed a strategy which can surprisingly encode a HDR scene (which is why we introduce the wording scene) actually as an LDR image (+ color transformation metadata), so whereas for simplicity of understanding various of our concepts and technical meta-structures may be elucidated with a scenario where Im_1, the image to be communicated to a receiving side is a HDR image, which should be re-gradable into an LDR image, the same principles are also useable, and to be used in other important market scenarios, in case Im_1 is actually an LDR grading (which can at a receiving side be re-graded into an HDR image, or any medium dynamic range image MDR, or any image outside the range of the communicated LDR and HDR gradings).
Applicant has generically taught on the concept of generating various additional gradings on the basis of available graded images in WO2012127401, which teaches the main concepts needed in display tuning, needed for all or at least a larger class of actual embodiments. However, it was still a problem to come up with simple coding variants, which conform to practical limitations of e.g. IC complexity, grader work capacity, etc., which the inventors could do after fixing some common ground principles for such practical HDR image handling. Another prior art document which is merely tangentially relevant is WO2013/046095, which merely teaches that there may be several different kinds of HDR coding coming in (i.e. with different PB_H values, e.g. a 5000 nit or only 2000 nit variant of the original HDR scene which was camera-captured), and various output displays to be served with various display peak brightness (PB_D) values, but again this merely teaches the technical framework needed, without going into all the details below. Correspondingly, there still was a problem of how to come with a matching simple technology for allowing the content creator to adjust the artistically optimized grading to the at least one actual present display at a receiving side.